Kernel Fisher’s Discriminant Analysis in Gaussian Reproducing Kernel Hilbert Space1

Kernel Fisher’s linear discriminant analysis (KFLDA) has been proposed for nonlinear binary classification (Mika, Rätsch, Weston, Schölkopf and Müller, 1999, Baudat and Anouar, 2000). It is a hybrid method of the classical Fisher’s linear discriminant analysis and a kernel machine. Experimental results (e.g., Schölkopf and Smola, 2002) have shown that the KFLDA performs slightly better in terms of prediction error than the popular support vector machines and is a strong competitor to the latter. However, there is very limited statistical justification of this method. In this article we provide a fundamental study for it in the framework of a Gaussian reproducing kernel Hilbert space (RKHS) and give an extension of the KFLDA to a quadratic generalization, called kernel Fisher’s quadratic discriminant analysis (KFQDA). In our approach each data point is mapped to a function in the kernel associated RKHS. Next the problem of Fisher’s discriminant is solved in this new kernel data space. This kernelized Fisher’s approach can be regarded, from the original data space viewpoint, as a nonparametric approach to classification since it adopts kernels as basis functions and its decision boundary is modeled via kernel mixture. Still it has the computational advantage of keeping the training process analogous to a parametric method. We show that the kernel Fisher’s discriminant can be obtained as a maximum likelihood method and a Bayes classifier under Gaussian assumption in the associated RKHS. We also demonstrate that our kernel transformations can drastically draw the data distribution to better Gaussian. Theorems are given to show that, under suitable conditions, most projections of data represented via kernel functions in a RKHS are approximately Gaussian, which justifies the Gaussian assumption in the RKHS.